Image processing apparatus and image processing method

ABSTRACT

An apparatus includes a first acquisition unit configured to acquire main object information specifying a main object in generation of a layout image, a second acquisition unit configured to acquire object correlation information specifying an object having a correlation with the main object, an extraction unit configured to extract at least one image including the main object and at least one image including the object having the correlation with the main object from a plurality of images based on the acquired main object information and the acquired object correlation information acquired, and a generation unit configured to generate, using a layout template, a layout image in which the at least one image extracted by the extraction unit and including the main object and the at least one image extracted by the extraction unit and including the object having the correlation with the main object are laid out therein.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.15/162,322, filed May 23, 2016, which is a Continuation of U.S. patentapplication Ser. No. 14/668,792, filed Mar. 25, 2015, now U.S. Pat. No.9,373,037, which is a Continuation of U.S. patent application Ser. No.13/934,400, filed Jul. 3, 2013, now U.S. Pat. No. 9,014,487, whichclaims the benefit of Japanese Application No. 2012-153670, filed Jul.9, 2012, all of which are hereby incorporated by reference herein intheir entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an apparatus and a method foroutputting a layout image including a predetermined object.

Description of the Related Art

Conventionally, methods have been known in which photographs taken witha digital camera are used to produce a variety of products such as analbum. Japanese Patent Application Laid-Open No. 2008-217479 discussesan image layout method including selecting a template and an image of atarget person, extracting the target person from an image database, andautomatically laying out the image according to an attribute of eacharea of the template.

However, the image layout method discussed in Japanese PatentApplication Laid-Open No. 2008-217479 can only generate a layout imagewith a focus on the target person. Hence, the image layout method has aproblem that variations of layout images that can be generated arelimited. Furthermore, the image layout method discussed in JapanesePatent Application Laid-Open No. 2008-217479 cannot generate a layoutimage that takes into consideration a relationship between the targetperson and other persons.

SUMMARY OF THE INVENTION

The present invention is directed to an apparatus and a method capableof overcoming the problems of the conventional techniques and outputtinga layout image in which a desired object is laid out as appropriate.

According to an aspect of the present invention, an apparatus includes afirst acquisition unit configured to acquire main object informationspecifying a main object in generation of a layout image, a secondacquisition unit configured to acquire object correlation informationspecifying an object having a correlation with the main object, anextraction unit configured to extract at least one image including themain object and at least one image including the object having thecorrelation with the main object from a plurality of images based on themain object information acquired by the first acquisition unit and theobject correlation information acquired by the second acquisition unit,and a generation unit configured to generate, using a layout template, alayout image in which the at least one image extracted by the extractionunit and including the main object and the at least one image extractedby the extraction unit and including the object having the correlationwith the main object are laid out.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a hardware configuration of an imageprocessing apparatus according to a first exemplary embodiment of thepresent invention.

FIG. 2 is a block diagram illustrating software of the image processingapparatus according to the first exemplary embodiment.

FIG. 3 is a flow chart illustrating image processing according to thefirst exemplary embodiment.

FIG. 4 illustrates a display example of an image group of each humanobject according to the first exemplary embodiment.

FIG. 5 illustrates an example of a user interface (UI) for setting mainobject information and object correlation information according to thefirst exemplary embodiment.

FIG. 6 illustrates an example of a layout template according to thefirst exemplary embodiment.

FIG. 7 illustrates a display example of a result of layout generationaccording to the first exemplary embodiment.

FIGS. 8A and 8B illustrate examples of object correlation informationaccording to second and third exemplary embodiments of the presentinvention.

FIGS. 9A and 9B illustrate examples of object correlation informationaccording to the second and third exemplary embodiments.

FIG. 10 illustrates an example of a storage format of a result of imageanalysis according to the first exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings. Asused herein, the terms “main object,” “main person,” and “main target”refer to the same meaning. The following exemplary embodiments are notintended to limit the scope of the invention set forth in the claims,and not every feature of combinations described in the exemplaryembodiments is always necessary for a technical solution of the presentinvention.

FIG. 1 is a block diagram illustrating an example of a hardwareconfiguration of an image processing apparatus according to a firstexemplary embodiment of the present invention.

In FIG. 1, an information processing apparatus 115 includes a centralprocessing unit (CPU) 100, a read only memory (ROM) 101, a random accessmemory (RAM) 102, a secondary storage device 103, a display device 104,an input device 105, an interface (IF) 107, an IF 108, and a wirelesslocal area network (LAN) 109. The information processing apparatus 115further includes an internal image capturing device 106. The foregoingcomponents are connected to one another via a control bus/data bus 110.The information processing apparatus 115 according to the presentexemplary embodiment functions as an image processing apparatus.

The information processing apparatus 115 is, for example, a computer.The CPU 100 executes information processing, which will be described inthe first exemplary embodiment, according to a program. The ROM 101stores programs including applications and operating systems (OS), whichwill be described below, that are to be executed by the CPU 100. The RAM102 provides a memory configured to store a variety of informationtemporarily at the time of execution of a program by the CPU 100. Thesecondary storage device 103 is a storage medium such as a hard diskconfigured to store a database that stores image files and results ofimage analysis. The display device 104 is a device such as a displayconfigured to present results of processing of the first exemplaryembodiment to a user. The display device 104 may possess a touch panelfunction. The input device 105 is a mouse or a keyboard with which auser inputs an instruction to execute image correction processing.

An image captured by the internal image capturing device 106 is storedin the secondary storage device 103 after predetermined imageprocessing. The information processing apparatus 115 can also read imagedata from an external imaging device 111 connected via an interface (IF108). The wireless LAN 109 is connected to the Internet 113. Theinformation processing apparatus 115 can also acquire image data from anexternal server 114 connected to the Internet 113.

A printer 112 configured to output images is connected to theinformation processing apparatus 115 via the IF 107. The printer 112 isalso connected to the Internet 113 and can transmit and receive printdata via the wireless LAN 109.

FIG. 2 is a block diagram illustrating a software configurationincluding an application according to the present exemplary embodiment.

Generally, image data acquired by the information processing apparatus115 is compressed in a compression format such as Joint PhotographicExperts Group (JPEG). Hence, an image codec unit 200 decompresses theimage data based on the compression format to convert the image datainto image data in a bitmap data format of a red-green-blue (RGB) dotsequential system (bitmap data). The converted bitmap data istransmitted to a display and an UI control unit 201 and displayed on thedisplay device 104 such as a display.

The bitmap data is also input into an image sensing unit 203(application), and the image sensing unit 203 executes a variety ofimage analysis processing, which will be described in detail below. Avariety of image attribute information obtained as a result of theanalysis processing is stored according to a predetermined format in thesecondary storage device 103 by a database unit 202 (application). Inthe present exemplary embodiment, the image attribute informationincludes main object information and object correlation information.Hereinafter, the terms image analysis processing and sensing processingwill be used interchangeably.

An object correlation information acquisition unit 209 (application)acquires the object correlation information stored in the database unit202. A main object information acquisition unit 208 (application)acquires the main object information stored in the database unit 202.

An image extraction unit 210 (application) extracts an image from thedatabase unit 202 based on the main object information and the objectcorrelation information.

A layout generation unit 205 (application) executes processing toautomatically generate a layout where image data is to be laid out byuse of the image extracted by the image extraction unit 210.

A rendering unit 206 renders the generated layout into display bitmapdata. The bitmap data, which is a rendering result, is transmitted tothe display and UI control unit 201, and contents of the bitmap data aredisplayed on the display device 104. The rendering result is alsotransmitted to a print data generation unit 207. The print datageneration unit 207 converts the rendering result into printer commanddata and transmits the converted printer command data to the printer112.

A flow of image processing is described in detail below with referenceto FIG. 3. FIG. 3 is a flow chart illustrating processing executed inthe software configuration illustrated in FIG. 2.

In step S1, the information processing apparatus 115 acquires imagedata.

In step S2, decoding processing of the acquired image data is executed.First, the image sensing unit 203 searches for newly-stored image datathat has not undergone the sensing processing yet. Then, the image codecunit 200 converts (decodes) each extracted image from image data(compressed image data) into bitmap data. The converted bitmap data istransmitted to the display and UI control unit 201 to be displayed onthe display device 104 such as a display.

In step S3, image sensing and database registration are executed.Specifically, the bitmap data is input into the image sensing unit 203,and the image sensing unit 203 executes a variety of analysisprocessing. A variety of image attribute information obtained as aresult of the analysis processing is stored according to a predeterminedformat in the secondary storage device 103 by the database unit 202.

In step S4, image grouping processing is executed. Specifically, theinput image is classified according to individual persons recognized bythe image analysis.

In step S5, main object information acquisition processing is executed.Specifically, the main object information acquisition unit 208 acquires,from the database unit 202, information on a human object to be set as amain object when a layout image is generated.

In step S6, the object correlation information acquisition unit 209acquires, from the database unit 202, information on a correlationbetween a set human object and a main person.

In step S7, the image extraction unit 210 extracts an appropriate imagefrom the database unit 202 based on the acquired main object informationand the object correlation information, i.e., the information on thecorrelation between the set human object and the main person.Specifically, the image extraction unit 210 extracts an image of a humanobject set in the object correlation information, with a focus on animage of a designated main person. The display and UI control unit 201controls the extracted image so that the display device 104 displays theextracted image.

In step S8, the layout generation unit 205 executes automatic layoutgeneration processing.

In step S9, rendering is executed. Specifically, the rendering unit 206renders the generated layout into display bitmap data.

In step S10, a layout image is displayed and/or printed based on therendering result. Specifically, the bitmap data obtained in step S9 istransmitted to the display and UI control unit 201, and the result isdisplayed on the display. The bitmap data is also transmitted to theprint data generation unit 207, and the print data generation unit 207converts the transmitted bitmap data into printer command data andtransmits the converted printer command data to the printer 112.

The following describes each processing in detail.

The acquisition of an image data group in step S1 is executed asfollows. For example, a user connects an image capturing apparatus or amemory card storing captured images to the information processingapparatus 115, and the information processing apparatus 115 reads thecaptured images from the image capturing apparatus or the memory card toacquire an image data group. Alternatively, the information processingapparatus 115 may acquire an image data group by reading images capturedby an internal image capturing device and stored in a secondary storagedevice. Further alternatively, the information processing apparatus 115may acquire images from an apparatus other than the informationprocessing apparatus 115, such as the external server 114 connected tothe Internet 113, via the wireless LAN 109.

The following describes the sensing processing (image analysisprocessing) executed in step S3. An application executes a variety ofanalysis processing and database registration of analysis results withrespect to each acquired image data group.

As used herein, the sensing processing includes a variety of processingspecified in Table 1. Examples of sensing processing in the presentexemplary embodiment include face detection, basic image featurequantity analysis, and scene analysis, which respectively providecalculation results of the data type specified in Table 1.

TABLE 1 Main class of Sub class of sensing sensing Data type Value Basicimage Average luminance int 0 to 255 feature quantity Average color int0 to 255 saturation Average hue int 0 to 359 Face Number of faces int 0to MAXFACE detection of human objects Coordinate int * 8 0 to Width orposition Height Average of Y in int 0 to 255 face region Average of Cbin int −128 to 127 face region Average of Cr in int −128 to 127 faceregion Scene Scene result char Landscape analysis Nightscape PortraitUnderexposure Others

The following describes each sensing processing.

The overall average luminance and the overall average color saturation,which are basic image feature quantities, may be calculated by, forexample, a publicly-known method. Thus, detailed description is omitted.The average luminance may be calculated by converting (conversionequation is omitted) RGB components of each pixel of an image intopublicly-known brightness/color-difference components (for example,YCbCr components) and then calculating an average value of the Ycomponent. The average color saturation may be calculated by calculatinga value of S for the CbCr components of each pixel using formula (1)below and then calculating an average value of S.S=√{square root over (Cb ² +Cr ²)}  (1)

The average hue (AveH) of an image is a feature quantity for evaluatingthe color tone of the image. The hue of each pixel can be calculatedusing a publicly-known hue-intensity-saturation (HIS) conversionequation, and AveH can be calculated by averaging the calculated hues ofthe entire image.

The foregoing feature quantities may be calculated for an entire image,or, for example, an image may be divided into regions of predeterminedsize, and the feature quantities may be calculated for each region.

The following describes detection processing of faces of human objects.Various publicly-known methods may be used as a method for the detectionof faces of human objects in the present exemplary embodiment.

In a method discussed in Japanese Patent Application Laid-Open No.8-63597, a matching level between an image and a plurality of templatesin the shape of a face is calculated. Then, a template with the highestmatching level is selected, and if the highest matching level is equalto or higher than a predetermined threshold value, then a region in theselected template is determined as a candidate face region. Thepositions of eyes can be detected using the template.

In a method discussed in Japanese Patent Application Laid-Open No.2000-105829, first, an entire image or a designated region of an imageis scanned using a nose image pattern as a template, and a position thatmost closely matches is output as a nose position. A region of the imageabove the nose position is considered to include eyes. Hence, aneye-existing region is scanned using an eye image pattern as a templateto execute matching, and a set of candidate eye-existing positions,which is a set of pixels with a higher matching level than apredetermined threshold value, is obtained. Then, continuous regionsincluded in the set of candidate eye-existing positions are separated asclusters, and the distance between each cluster and the nose position iscalculated. A cluster with the shortest distance from the nose positionis determined as an eye-existing cluster, whereby the position of theorgan is detected.

Examples of other methods for the detection of faces of human objectsinclude methods of detecting positions of faces and organs discussed inJapanese Patent Application Laid-Open No. 8-77334, Japanese PatentApplication Laid-Open No. 2001-216515, Japanese Patent ApplicationLaid-Open No. 5-197793, Japanese Patent Application Laid-Open No.11-53525, Japanese Patent Application Laid-Open No. 2000-132688,Japanese Patent Application Laid-Open No. 2000-235648, and JapanesePatent Application Laid-Open No. 11-250267. A method discussed inJapanese Patent No. 2541688 may also be used. The method for thedetection of faces of human objects is not particularly limited.

The feature quantities of a face region can be analyzed by the facedetection processing of human objects. For example, the number of facesof human objects and the coordinate position of each face can beobtained for each input image. Since the coordinate positions of thefaces in the image are obtained, average values of YCbCr components ofpixels included in each face region can be calculated to obtain theaverage luminance and the average color difference of each face region.

Scene analysis processing can be executed using the feature quantitiesof images. Scene analysis processing can be executed by, for example, amethod discussed in Japanese Patent Application Laid-Open No.2010-251999 or Japanese Patent Application Laid-Open No. 2010-273144. Asa result of the scene analysis, identifications (IDs) for discriminatingimage-captured scenes such as landscape, nightscape, portrait,underexposure, and others are obtained.

Although the sensing information is obtained by the sensing processingin the present exemplary embodiment, the present invention is notlimited to the present exemplary embodiment, and other sensinginformation may also be used.

The obtained sensing information described above is stored in thedatabase unit 202. The format of storage in the database is notparticularly limited. For example, the sensing information may bewritten in a general format (for example, extensible markup language(XML)) and stored.

The following describes an exemplary case in which attribute informationfor each image is written in three separate categories as illustrated inFIG. 10.

The first tag, BaseInfo tag, is a tag for storing information that isadded to an acquired image file in advance as an image size andinformation on the time of image capturing. The tag includes anidentifier ID of each image, storage location at which image files arestored, image size, and information obtained at the time of imagecapturing such as a place of image capturing, time, in-focus position,and presence or absence of a flash.

The second tag, SensInfo tag, is a tag for storing the results of imageanalysis processing. The tag stores the average luminance, average colorsaturation, average hue, and scene analysis results of an entire image.The tag also stores information on human objects existing in images,face position, face size, number of faces, and face complexion.

The third tag, UserInfo tag, is a tag for storing information indicatingthe favorite degree that is input by a user for each image and historyinformation on the usage of images such as the number of times ofprinting and viewing through an application and the number of times oftransmissions through the Internet.

The method for the database storage of image attribute information isnot limited to the foregoing method, and the image attribute informationmay be stored in any other format.

The following describes the image grouping processing executed in stepS4. In step S4, identical human objects are recognized using thedetected face information to generate an image group for each humanobject.

A method of executing recognition of human objects is not particularlylimited. For example, a publicly-known method for the recognition ofindividual persons may be used to execute the recognition of humanobjects. Recognition processing of individual persons is executed mainlyby extracting feature quantities of organs existing within a face suchas eyes and a mouth and comparing similarity levels of relationship ofthe feature quantities. A specific method of recognition processing ofindividual persons is discussed in, for example, Japanese Patent No.3469031 and elsewhere. Thus, detailed description is omitted.

Referring back to FIG. 3, the image grouping processing executed in stepS4 is described below.

In the image grouping processing, feature quantities of faces includedin an image are calculated, and images with similar feature quantitiesare grouped as face images of the same human object to give the samehuman object identifier (ID). As used herein, the feature quantities offaces include the positions and sizes of organs such as eyes, a mouth,and a nose, and a facial contour. A face image that has been given an IDis written in a person tag of the image.

The image group of each human object obtained by the foregoingprocessing is displayed on the display device 104. In the presentexemplary embodiment, the image group is displayed on a UI 1501illustrated in FIG. 4. In FIG. 4, a region 1502 displays arepresentative face image of the image group of the human object. Aregion 1503 next to the region 1502 displays a name of the human object(“father” in this case). A region 1504 displays thumbnails of faceimages of images included in the image group. Specifically, the region1504 displays a plurality of face images recognized as including thehuman object.

When a human object (“son”) other than “father” is recognized, an imagegroup of face images including the son is displayed as in the foregoingcase.

Information on each human object can be input via an input unit on theUI 1501. For example, a birthday can be input via a first input unit1505, and relationship information can be input via a second input unit1506.

The following describes the main object information acquisitionprocessing executed in step S5 and the object correlation informationacquisition processing executed in step S6. In the main objectinformation acquisition processing executed in step S5, information on ahuman object to be prioritized at the time of image extraction (mainobject information) is acquired. In the object correlation informationacquisition processing executed in step S6, information on an objecthaving a correlation with the main person is acquired with respect tothe main object information determined in step S5. For example,information on an object having a close relation with the main person isacquired. The object correlation information specifies a human object tobe prioritized next to the main person at the time of image extraction.The following describes a method of determining the main objectinformation and the object correlation information with reference toFIG. 5. FIG. 5 is a view illustrating a user interface for determiningthe main object information and the object correlation information. Thisuser interface is displayed on, for example, the display device 104.

In FIG. 5, a work area 3401 is a display area for displaying a varietyof information at the time of execution of an application, prompting auser to select, and showing a preview. A work area 3402 is an area fordisplaying a recognized human object. A work area 3403 is an area fordisplaying a button for various operations.

The work area 3402 displays a representative image of a human objectrecognized as individual persons in steps S3 and S4. For example, when“father,” “son,” and “friend” are recognized as individual persons, workareas 3404, 3405, and 3406 in the work area 3402 display representativeimages of “father,” “son,” and “friend,” respectively, as illustrated inFIG. 5. The work area 3402 includes menu buttons 3407, 3408, 3409 forsetting the main object information and the object correlationinformation for each human object.

A user can set the main object information and the object correlationinformation by operating the menu buttons. For example, when “son” isset to “main” as a human object to be prioritized and “father” is set to“sub” as a human object to be prioritized next to the main object, “son”is set as the main object information and “father” is set as the objectcorrelation information. When “friend” is set to “sub” in place of“father,” “friend” is set as a human object to be prioritized next tothe main person.

In the present exemplary embodiment, a single object is settable foreach of the “main” object and the “sub” object. However, the presentinvention is not limited to the present exemplary embodiment, and aplurality of objects may be set for each of the “main” object and the“sub” object.

The method of setting the main object information and the objectcorrelation information is not limited to the foregoing method. Forexample, the main object information and the object correlationinformation may be set based on main object information and objectcorrelation information stored in advance in a storage device. Theobject correlation information may be information on a correlation levelbetween each personal ID and other personal IDs (order of closeness) orinformation on grouping such as “family,” “school,” and “company.”

In step S7, appropriate images are extracted based on the acquired mainobject information and the acquired object correlation information.Specifically, with a focus on images of the designated main person, atleast one image of each human object considered to have a close relationwith the main person is extracted. For example, the percentage of imagesincluding the main person (main object) in all images extracted at thetime of image extraction may be set to a predetermined percentage orhigher. Similarly, the percentage of images including an object having acorrelation with the main person (main object) in all images extractedat the time of image extraction may be set to a predetermined percentageor higher. For example, the total of the percentage of images includingthe main object in all extracted images and the percentage of imagesincluding an object having a correlation with the main object in allextracted images may be set to 50% or higher. The percentage of imagesincluding the main object in all extracted images may be set to 30% orhigher, and the percentage of images including an object having acorrelation with the main object may be set to 20% or higher. At thistime, an image including both the main object and an object having acorrelation with the main object may be counted not as an imageincluding an object having a correlation with the main object but as animage including the main object. Alternatively, an image including boththe main object and an object having a correlation with the main objectmay be counted not as an image including the main object but as an imageincluding an object having a correlation with the main object. Imagesmay be filtered to extract only images including the main person andimages including an object having a correlation with the main person.Images may also be filtered not to extract images including a humanobject considered to have little relation with the main person.

Images may also be filtered such that while images including a humanobject having little or no relation with the main person are notextracted as an image of a human object, images including no humanobject such as a landscape are extracted. The percentage of imagesincluding the main object and the percentage of images including anobject having a relation with the main object may be set by a user asappropriate or may be set to predetermined percentages in advance. Theextracted images are displayed on the display device 104 by the displayand UI control unit 201.

The following describes the layout generation processing executed instep S8. In the present exemplary embodiment, the layout generationprocessing is executed using a variety of layout templates prepared inadvance. Examples of layout templates include a layout template in whicha plurality of image layout frames are provided on a layout image. Thepresent exemplary embodiment employs a layout template in which aplurality of image layout frames 1702, 1703, and 1704 are provided on asheet scale for a layout as illustrated in FIG. 6. Hereinafter, theimage layout frames are also referred to as slots. The page size (forexample, “A4”) and the page resolution (for example, “300 dpi”) are setas basic information for each layout template. Positional informationand shape information (for example, “rectangle”) are set for each slot.The layout template may be acquired from, for example, layout templatesstored in advance in the secondary storage device 103 at the time ofinstallation of software for the execution of the present exemplaryembodiment into the information processing apparatus 115. Alternatively,a template group may be acquired from the external server 114 existingon the Internet 113 connected via the IF 107 or the wireless LAN 109.

A layout is generated using a combination of information on a determinedtheme of the layout to be generated and a determined template,information on the main person, information on a correlation with themain person, and information on a set of selected images to be used togenerate the layout. The theme of the layout determines an outline ofthe layout, and examples include a growth record, a wedding ceremony, atrip, and a graduation ceremony. In the present exemplary embodiment,one or more appropriate layout templates for each layout theme areprepared. Based on the foregoing information, image data to be used isselected from the set of image data and laid out to generate the layout.When the number of extracted images is fewer than the number of slotsincluded in the layout template, the template may be changed. A methodof generating a layout is not particularly limited, and examples includea method in which an image characteristic of an image to be laid out isdetermined in advance for each slot, and an image matching thedetermined image characteristic is selected and laid out. The imagecharacteristic is, for example, information obtained by the analysisprocessing such as a specific captured human object, the number of humanobjects, image brightness, image capturing information such asphotographed time, and usage information such as print frequency.

For example, the image characteristics “an image of the main person, abright image, and an image with the face at the center” are designatedfor the slot 1702. The image characteristics “an image of the mainperson and an image of the human object having a close relation with themain person” are designated for the slot 1703. The image characteristic“landscape” is designated for the slot 1704. “Son” is set as the mainperson, and “father” is set as a human object having a close relationwith the main person. Accordingly, in the layout generation processing,an image satisfying the condition designated for each slot is selectedand laid out. Specifically, an image of “son” is laid out in the slot1702. An image including both “son” and “father” is laid out in the slot1703. A landscape image including neither “son” nor “father” is laid outin the slot 1704.

The method of generating a layout is not limited to the foregoingmethod, and other examples include a method including generating a largenumber of layouts with extracted images being laid out, evaluating thegenerated layouts according to a given function, and determining alayout from upper ranked layouts. The evaluation may be, for example, acomprehensive evaluation based on a plurality of criteria such as imagecharacteristics, degree of matching in shape with a slot, layoutbalance, and conformity with the theme.

It is more suitable to use attribute information on the main person atthe time of generation and evaluation of layouts. Use of attributeinformation on the main person enables trimming to obtain a close-up ofa face of the main person and also enables selection of an image inwhich a face of the main person appears with appropriate brightness.When an image including both the main person and a human object having aclose relation with the main person is requested, an image in which thein-focus position matches the positions of the main person and the humanobject is selected by reference to the image capturing information. Acombination of a variety of information enables generation of betterlayouts.

In the present exemplary embodiment, images can be extracted with afocus on images including the main person and images including a humanobject having a relation with the main person. Thus, a layout imageincluding images laid out with a focus on images including the mainperson and images including a human object having a relation with themain person can be generated without designating an image characteristicof an image to be laid out for each slot. Furthermore, a layout imageincluding images laid out with a focus on images including the mainperson and images including a human object having a relation with themain person can be generated by only designating a human object or alandscape as an image characteristic for each slot without designatingdetails of the human object.

The generated layout is rendered using a rendering function of an OSoperating on the information processing apparatus 115 and displayed onthe UI. In the present exemplary embodiment, a region 2902 asillustrated in FIG. 7 is displayed. FIG. 7 includes the region 2902 fordisplaying the generated layout and various execution buttons includinga previous button 2903, a next button 2904, and a print button 2905.

Another layout can be presented in response to a user operation ofpressing the next button 2904. In other words, the user can view avariety of layouts by pressing the next button 2904. The user can pressthe previous button 2903 to redisplay a layout that was previouslydisplayed. When the user likes a displayed layout, the user can pressthe print button 2905 to print out the layout result from the printer112 connected to the information processing apparatus 115.

In the present exemplary embodiment, as described above, images areextracted based on the main object information and the objectcorrelation information so that a layout image including images laid outwith a focus on not only the main person but also one or more otherhuman objects having a relation with the main person can be obtained.

In other words, not only a target human object (main person) but also ahuman object having a relation with the target human object can beselected to generate a desired layout image. This image processingmethod is effective especially when, for example, a layout image as agift for a human object having a relation with the main person of thetheme is desired to be obtained.

Advantages of the present exemplary embodiment will be described brieflybelow using a wedding ceremony as an example. In a scene where a largenumber of images are captured such as a wedding ceremony, there are alarge number of human objects such as relatives and guests besides agroom and a bride who are main persons. At this time, human objectshaving little relation such as a priest and floor attendants are oftenphotographed together although they are not intended to be photographed.However, when face recognition processing is executed with respect tothe images, human objects included as small figures in the images suchas floor attendants are also picked up as human objects existing in theimages.

If a normal method of generating a layout image is used to generate alayout image of the wedding ceremony, unintended human objects may belaid out in the layout image. For example, in a method discussed inJapanese Patent Application Laid-Open No. 2008-217479, if “bride” isselected as a target human object (main person) to extract images, onlya layout image with a focus on the bride can be generated.

In contrast, the present exemplary embodiment enables a user to obtain alayout image for each purpose of use with ease such as a gift for afriend of the bride or a gift for a relative of the groom. For example,a layout image with a focus on the bride and friends of the bride can begenerated with ease by selecting “bride” as the main person and “friend”as a human object having a relation with the main person. At this time,two or more human objects may be set as friends. Further, a layout imagewith a focus on the groom and the grandmother of the groom can begenerated with ease by selecting “groom” as the main person and“grandmother” as a human object having a relation with the main person.To generate different layouts for different main persons, differentpurposes of use, or different viewers from the same set of images, themain object information acquisition processing in step S5 and thesubsequent processing may be repeated, whereby a user can obtain anappropriate layout simply by determining the main person and a humanobject having a relation with the main person.

A second exemplary embodiment of the present invention is similar to thefirst exemplary embodiment, except for the method of setting the objectcorrelation information. Thus, duplicate description of similar aspectsis omitted. In the present exemplary embodiment, the object correlationinformation is information on a relationship that is registered inadvance for each human object.

FIGS. 8A and 8B are views illustrating the object correlationinformation. FIG. 8A is a view illustrating human objects detected andrecognized from a plurality of photographs captured in a weddingceremony in the present exemplary embodiment. In FIG. 8A, 12 humanobjects are detected and recognized from the plurality of photographs.Each of the 12 human objects is given a personal ID (1, 2, . . . 12),and a name (A, B, C . . . ) and a relationship (bride, groom, mother ofA . . . ) are input. Affiliation information for grouping related humanobjects is added to each human object. For example, the human object Aof ID=1 is set as belonging to a family of A 3501, a school of A 3502,and a company of A and B 3503. The human object B of ID=2 is set asbelonging to a family of B 3504 and the company of A and B 3503. Thehuman object C of ID=3 is set as belonging to the family of A 3501. Theaffiliation information is set for every human object. In this case,there may be a human object with affiliation=none such as the humanobjects K and L.

As a result of grouping the above 12 human objects into groups ofrelated human objects, several affiliations (3501 to 3504) are formed asillustrated in FIG. 8B.

The affiliation information may be determined automatically frominformation input by a user such as relationships, names, and humanobject profiles. Alternatively, a user may perform grouping to manuallyset the affiliation information. A method of setting the affiliationinformation is not limited to the foregoing method. For example, theaffiliation information may be determined based on image attributeinformation that is not intentionally input by a user, such as imageanalysis information, image capturing information, and usageinformation.

Examples of image analysis information include results of basic imagefeature quantity analysis such as brightness, color saturation, and hueof images, information on human objects existing in images, results offace analysis such as the number of faces, positions, face size, andface complexion, and results of scene analysis.

Image capturing information is information obtained at the time ofcapturing an image such as a place and a time of image capturing, animage size, an in-focus position, and presence or absence of a flash.

Usage information is history information on the usage of images such asthe number of times of printing images, the number of times ofdisplaying images, and the number of times of transmitting imagesthrough the Internet.

Based on the foregoing information, human objects can be classified intoaffiliations according to various criteria. Examples of criteriainclude: the human objects are/are not photographed together; the focusis the same/different; the captured time is close; the human objects arephotographed by the same camera; the human objects are photographed inthe same event; and the human objects are printed together. For example,if the human objects are often photographed together, then the humanobjects are classified into the same affiliation. If the human objectsare not photographed together, then the human objects are classifiedinto different affiliations. Even when the human objects arephotographed together, if the focus is different, then the human objectsare understood as being photographed together by chance and are, thus,classified into different affiliations. If, for example, image data hasbeen accumulated for a long period of time, the human objects existingin images captured at close timings are determined to be the sameaffiliation. Alternatively, an affiliation may be determined based on acombination of a variety of information described above.

The following describes a method of determining the object correlationinformation according to the present exemplary embodiment using, as anexample, a case of generating a layout for grandparents of A and alayout for grandparents of B.

In the case of the relationships specified in FIGS. 8A and 8B, when alayout for the grandparents of A is intended to be generated, “personalID=1,” which is A, is set as the main object information. Following thedetermination of the main object, “family of A, school of A, company ofA and B,” which are affiliations to which A belongs, are determined asthe object correlation information from the affiliation information onthe main object. When a layout for the grandparents of B is intended tobe generated, “personal ID=2” of B is set as the main objectinformation. In this case, similarly, “family of B, company of A and B,”which are affiliations to which B belongs, are determined as the objectcorrelation information from the affiliation information on the mainobject.

As in the present exemplary embodiment, when the object correlationinformation is determined in advance, correlation information on anobject having a relation with the main object can be determined bysimply switching the main object information. Thus, a user can determinethe object correlation information without determining the objectcorrelation information via the UI. This enables the image extractionunit 210 to extract appropriate images for the purpose of use for eachuser at the time of image extraction.

The object correlation information is not limited to the informationdetermined by the method of classifying human objects into affiliationsas in the above case. For example, the object correlation informationmay be determined based on information specifying correlations betweenhuman objects. Specifically, as illustrated in FIGS. 9A and 9B, theobject correlation information may be determined based on information oncorrelations between each individual human object having an ID and otherhuman objects. In FIG. 9A, the human object A of ID=1 has a correlationwith the human objects of IDs=2, 3, 4, 5, 6, and 9. The human object Bof ID=2 has a correlation with the human objects of IDs=1, 7, 8, and 10.The human object C of ID=3 has a correlation with the human objects ofIDs=1, 4, and 5, and so on. The object correlation information isdetermined for every human object. In this case, there may be a humanobject having no correlation with any of the human objects, such as Kand L. The object correlation information is determined based on acorrelation ID set for each personal ID.

As in the case described with reference to FIGS. 8A and 8B, the objectcorrelation information may be determined automatically from user inputinformation, may be determined manually by a user, or may be determinedautomatically from information that is not intentionally input by a usersuch as image analysis information, image capturing information, andusage information.

The following describes a method of determining the object correlationinformation using, as an example, a case of generating a layoutincluding the object correlation information as illustrated in FIGS. 9Aand 9B.

As to a layout for the grandparents of A, if “personal ID=1,” which isA, is set as the main object information, “2, 3, 4, 5, 6, and 9,” whichare correlation IDs of A, may be determined as the object correlationinformation. As to a layout for the grandparents of B, if “personalID=2,” which is B, is set as the main object information, “1, 7, 8, and10,” which are correlation IDs of B, may be set as the objectcorrelation information.

In the present exemplary embodiment, information on the relationship isstored in advance for each registered human object, and the objectcorrelation information on an object having a correlation with the mainobject is determined based on the stored information. This allows theobject correlation information to be determined with ease each time whenthe main object information is switched, i.e., when the main object ischanged. Thus, related images can be obtained with ease.

In the present exemplary embodiment, every affiliation to which the mainobject belongs may be selected as the object correlation information, orevery human object determined as having a correlation with the mainobject may be selected as the object correlation information. Thisenables easy extraction of every human object having a correlation.

Traditionally, sorting of images for each human object requires a lot ofwork. In contrast, in the present exemplary embodiment classification ofhuman objects based on the main object information and the objectcorrelation information does not require a lot of work and can beconducted with ease. Thus, a desired image can be extracted with easesimply by setting the object correlation information.

A third exemplary embodiment of the present invention is similar to thefirst exemplary embodiment, except for the method of setting the objectcorrelation information. Thus, duplicate description of similar aspectsis omitted. Compared to the second exemplary embodiment, the presentexemplary embodiment further limits human objects having a correlationwith a main person in setting the object correlation information. Thefollowing describes the present exemplary embodiment using the case ofrelationships illustrated in FIGS. 8A and 8B as an example.

As illustrated in FIG. 8A, the human object A belongs to the followingaffiliations: family of A, school of A, and company of A and B. Althoughthe human object A (main object information: ID=1) is determined as themain person at the time of layout generation, appropriate images differdepending on the purpose of layout generation and viewers. For example,when a layout image for a grandmother of A is generated, “family of Aaffiliation 3501” is considered to be appropriate as the objectcorrelation information. Therefore, “family of A affiliation 3501” isset as the object correlation information. When a layout image for usein an in-house magazine of the company of A is generated, since it isnot appropriate to use a photograph including a human object who isirrelevant to the company, “company of A and B 3503” is considered to beappropriate as the object correlation information. Therefore, “companyof A and B 3503” is set as the object correlation information.

Once the object correlation information is set as described above,images of the human objects of IDs=3, 4, and 5, who are members of thefamily of A, are extracted besides the main person A (ID=1) in the caseof generating the layout image for the grandmother of A. In the case ofgenerating the layout image for use in an in-house magazine of thecompany of A, photographs of the human objects of IDs=2, 9, and 10, whoare members of the company of A and B, are extracted besides the mainperson A (ID=1).

As in the foregoing cases, images can be extracted more appropriately bylimiting the affiliation to be used among the plurality of affiliationsto which the main object belongs.

A user can set an affiliation to be used as the object correlationinformation. In the case of relationships illustrated in FIGS. 9A and9B, images can be extracted more appropriately by selecting a humanobject from correlated human objects. For example, in the case ofgenerating the layout image for the grandmother of A, “IDs=2, 3, 4, and5” are considered to be appropriate as the object correlationinformation. Thus, “IDs=2, 3, 4, and 5” are set as the objectcorrelation information. In the case of generating the layout image foruse in an in-house magazine of the company of A, “IDs=2 and 9” areconsidered to be appropriate as the object correlation information.Thus, “IDs=2 and 9” are set as the object correlation information. Auser can select a human object to be set as the object correlationinformation.

Compared with the second exemplary embodiment, the present exemplaryembodiment can select a more appropriate human object as a human objecthaving a correlation with the target human object (main person) to setthe selected human object as the object correlation information.

Although the foregoing describes each exemplary embodiment of thepresent invention, the basic configuration of the present invention isnot limited to the above exemplary embodiments.

In the above exemplary embodiments, a user intentionally determines themain object information. Alternatively, the main object information maybe determined automatically from information that is not intentionallyinput by the user, such as image analysis information, image capturinginformation, and usage information. From the foregoing information, themain object information may be determined based on criteria. Examples ofcriteria include: the human object appears in the largest number ofimages; the human object appears in the largest total area; there aremany close-ups of the face of the human object; the image captured timeis largely dispersed (the human object appears evenly); the human objectappears in images included in a registered event; and a large number ofimages of the human object are printed.

The main object information and the object correlation information foruse in the layout generation may be determined from layout information.Examples of layout information include a layout theme, timing of layoutgeneration, and a layout target period. The layout target period is tolimit the timing at which images to be extracted were captured, such asa trip and an event. For example, when an application determines thatthe first birthday of a child of a user is coming soon and suggests theuser to generate a layout with the theme of a growth record, “child” maybe set as the main object information, and “family” may be set as theobject correlation information. When the theme is set to “weddingceremony,” human objects named “groom” or “bride” may be set as the mainobjects.

Although the foregoing describes the cases of generating the layoutbased on the main object information and the object correlationinformation, the human object display screen illustrated in FIG. 4 mayreflect the main object information and the object correlationinformation. When the main object information and the object correlationinformation are reflected on the display screen, only the main personand human objects having a close relation with the main person aredisplayed. This enables the user to see the relations with ease andconfirm images classified for each human object.

The main object is not limited to a single object, and both the groom(ID=2) and the bride (ID=1) may be set as the main objects.

In the present exemplary embodiment, the affiliations are handledequally, but a plurality of affiliations may be given a priority orderaccording to closeness to the main person. Further, although the humanobjects belonging to the same affiliation are handled equally in thepresent exemplary embodiment, each of the human objects in anaffiliation may be given a priority order. For example, as to thepriority order of the family of A, the priority order of the father of A(ID=4) and the mother of A (ID=3) may be set higher, while the priorityorder of the aunt of A (ID=5) may be set lower. When a priority order isgiven to the affiliations or to the human objects, a weight may be givenaccording to the priority order. As described above, when the priorityorder or the weight is given, the priority order or the weight may betaken into consideration at the time of image extraction or at the timeof image layout. For example, images of a human object given a highpriority order or weight may be extracted more than other images at thetime of image extraction. Further, images of a human object with a highpriority order or weight may be laid out near the center or in a slotthat is large and eye-catching at the time of image layout.

In the foregoing exemplary embodiment, a layout image is obtained bylaying out a plurality of images on a layout template including aplurality of image layout frames. However, the layout template is notlimited to that used in the exemplary embodiment. For example, thelayout template may be a layout template in which image layout referencepoints are provided on a layout surface. The image layout referencepoints may be provided on the layout surface, and images may be laid outon the layout surface such that image layout points and each imagepartly correspond, e.g., image layout points and image reference pointsprovided to each image are associated. In the above exemplaryembodiment, an appropriate layout template is determined according tothe layout theme. However, the present invention is not limited to theexemplary embodiment. For example, a user may determine a layouttemplate. In this case, the layout generation unit may automaticallygenerate a layout image by laying out a plurality of images extracted bythe image extraction unit on the layout template determined by the user.

Although the above exemplary embodiments are described using the case ofusing a human object as the object, the object is not limited to a humanobject. The recognition processing of a pet such as a dog and a cat maybe executed to recognize the pet so that the pet can be set as anobject. The recognition processing may be executed to recognize a shapesuch as edge detection to recognize a building and a small article sothat the building and the small article can be set as an object. Ifregistration as an object is successful, the object correlationinformation can also be set. For example, when a dog that is a pet of amain person is set as the object correlation information, imagesincluding the main object together with the dog that is a pet can beextracted.

According to the exemplary embodiment of the present invention, a layoutimage in which a desired object is laid out as appropriate can be outputwith ease. The present exemplary embodiment can output a layout imageappropriate for the purpose of use of the user so that the user canobtain a highly satisfactory suitable layout image.

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)®),a flash memory device, a memory card, and the like. In addition, theentire processing is not necessarily realized by software, and a part ofthe processing or the entire processing may be realized by hardware.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass such all modifications and equivalent structures andfunctions.

What is claimed is:
 1. An image processing apparatus, comprising: atleast one processor coupled to at least one memory, the at least oneprocessor configured to perform operations comprising: acquiring aplurality of images; executing an analyzing process includingrecognizing objects included in the images; causing a display to displaya screen including a setting section; and executing a layout process forgenerating a layout image based on the analyzing process and a user'sinput on the setting section, wherein a first face image of a firstobject and a second face image of a second object are displayed on thesetting section, the first object and the second object having beenrecognized in the analyzing process, so that a user is able to set, onthe setting section, a first priority in the layout process to the firstobject and able to set a second priority lower than the first priorityin the layout process to the second object, while the first face imageand the second face image are displayed on the setting section, andwherein the layout process is executed based on a result of setting thefirst priority and the second priority on the setting section.
 2. Theimage processing apparatus according to claim 1, wherein, on the settingsection, the first priority related to adopting an image in the layoutprocess is able to be set to the first object and the second priorityrelated to adopting an image in the layout process is able to be set tothe second object, whereby an image including the first object and animage including the second object are preferentially adopted in thelayout process.
 3. The image processing apparatus according to claim 1,wherein an image including the first object and an image including thesecond object are preferentially adopted in the layout process so that alarger number of images including the first object and images includingthe second object are laid out in the layout process than an imageincluding neither the first object nor the second object.
 4. The imageprocessing apparatus according to claim 1, wherein the image includingthe first object is adopted more preferentially than the image includingthe second object in the layout process.
 5. The image processingapparatus according to claim 1, wherein the image including the firstobject is adopted more preferentially than the image including thesecond object in the layout process so that the image including thefirst object is laid out in a larger slot included in a template thanthe image including the second object in the layout process.
 6. Theimage processing apparatus according to claim 1, wherein the imageincluding the first object is adopted more preferentially than the imageincluding the second object in the layout process so that a largernumber of images including the first object is laid out than imagesincluding the second object in the layout process.
 7. The imageprocessing apparatus according to claim 1, wherein the first priority isable to be set to a plurality of objects.
 8. The image processingapparatus according to claim 1, wherein an image feature quantity isacquired in the analyzing process.
 9. The image processing apparatusaccording to claim 1, wherein the analyzing process includes sceneanalysis.
 10. The image processing apparatus according to claim 1,wherein a template to be used in the layout process is selectedaccording to a theme of a layout image to be generated.
 11. The imageprocessing apparatus according to claim 1, wherein a person isrecognizable as an object in the analyzing process.
 12. The imageprocessing apparatus according to claim 11, wherein at least one of adog and a cat is further recognizable as an object in the analyzingprocess.
 13. The image processing apparatus according to claim 12,wherein preferential adoption of an image including the recognizedperson and the at least one of the recognized dog and the recognized catin the layout process is settable on the setting section.
 14. The imageprocessing apparatus according to claim 1, wherein the layout process isexecuted automatically.
 15. The image processing apparatus according toclaim 1, wherein the layout image is to be converted to printable printdata.
 16. An image processing method, comprising: acquiring a pluralityof images; executing an analyzing process including recognizing objectsincluded in the images; causing a display to display a screen includinga setting section; and executing a layout process for generating alayout image based on the analyzing process and a user's input on thesetting section, wherein a first face image of a first object and asecond face image of a second object are displayed on the settingsection, the first object and the second object having been recognizedin the analyzing process, so that a user is able to set, on the settingsection, a first priority in the layout process to the first object andable to set a second priority lower than the first priority in thelayout process to the second object, while the first face image and thesecond face image are displayed on the setting section, and wherein thelayout process is executed based on a result of setting the firstpriority and the second priority on the setting section.
 17. The imageprocessing method according to claim 16, wherein, on the settingsection, the first priority related to adopting an image in the layoutprocess is able to be set to the first object and the second priorityrelated to adopting an image in the layout process is able to be set tothe second object, whereby an image including the first object and animage including the second object are preferentially adopted in thelayout process.
 18. The image processing method according to claim 16,wherein an image including the first object and an image including thesecond object are preferentially adopted in the layout process so that alarger number of images including the first object and images includingthe second object are laid out in the layout process than an imageincluding neither the first object nor the second object.
 19. The imageprocessing method according to claim 16, wherein the image including thefirst object is adopted more preferentially than the image including thesecond object in the layout process.
 20. The image processing methodaccording to claim 16, wherein the image including the first object isadopted more preferentially than the image including the second objectin the layout process so that the image including the first object islaid out in a larger slot included in a template than the imageincluding the second object in the layout process.
 21. The imageprocessing method according to claim 16, wherein the image including thefirst object is adopted more preferentially than the image including thesecond object in the layout process so that a larger number of imagesincluding the first object is laid out than images including the secondobject in the layout process.
 22. The image processing method accordingto claim 16, wherein the first priority is able to be set to a pluralityof objects.
 23. The image processing method according to claim 16,wherein an image feature quantity is acquired in the analyzing process.24. The image processing method according to claim 16, wherein theanalyzing process includes scene analysis.
 25. The image processingmethod according to claim 16, wherein a template to be used in thelayout process is selected according to a theme of a layout image to begenerated.
 26. The image processing method according to claim 16,wherein a person is recognizable as an object in the analyzing process.27. The image processing method according to claim 16, wherein at leastone of a dog and a cat is further recognizable as an object in theanalyzing process.
 28. The image processing method according to claim16, wherein preferential adoption of an image including the recognizedperson and the at least one of the recognized dog and the recognized catin the layout process is settable on the setting section.
 29. The imageprocessing method according to claim 16, wherein the layout process isexecuted automatically.
 30. The image processing method according toclaim 16, wherein the layout image is to be converted to printable printdata.